Search for: All records

Egocentric and exocentric perspectives of human action differ significantly, yet overcoming this extreme viewpoint gap is critical in augmented reality and robotics. We propose VIEWPOINTROSETTA, an approach that unlocks large-scale unpaired ego and exo video data to learn clip-level viewpoint-invariant video representations. Our framework introduces (1) a diffusion-based Rosetta Stone Translator (RST), which, leveraging a moderate amount of synchronized multi-view videos, serves as a translator in feature space to decipher the alignment between unpaired ego and exo data, and (2) a dual encoder that aligns unpaired data representations through contrastive learning with RST-based synthetic feature augmentation and soft alignment. To evaluate the learned features in a standardized setting, we construct a new cross-view benchmark using Ego-Exo4D, covering cross-view retrieval, action recognition, and skill assessment tasks. Our framework demonstrates superior cross-view understanding compared to previous view-invariant learning and ego video representation learning approaches, and opens the door to bringing vast amounts of traditional third-person video to bear on the more nascent first-person setting. 

We investigate exocentric-to-egocentric cross-view translation, which aims to generate a first-person (egocentric) view of an actor based on a video recording that captures the actor from a third-person (exocentric) perspective. To this end, we propose a generative framework called Exo2Ego that decouples the translation process into two stages: high-level structure transformation, which explicitly encourages cross-view correspondence between exocentric and egocentric views, and a diffusion-based pixel-level hallucination, which incorporates a hand layout prior to enhance the fidelity of the generated egocentric view. To pave the way for future advancements in this field, we curate a comprehensive exo-to-ego cross-view translation benchmark. It consists of a diverse collection of synchronized ego-exo tabletop activity video pairs sourced from three public datasets: H2O, Aria Pilot, and Assembly101. The experimental results validate that Exo2Ego delivers photorealistic video results with clear hand manipulation details and outperforms several baselines in terms of both synthesis quality and generalization ability to new actions. 

Abstract Vast amounts of carbon are stored beneath the seafloor in the form of methane hydrate. Hydrate is stable at moderate pressure and low temperature at a depth extending several hundred meters beneath the seafloor to the base of gas hydrate stability (BGHS) often marked by bottom simulating reflections (BSRs) in seismic profiles. However, data from logging‐while‐drilling and coring during Integrated Ocean Discovery Program Expeditions 372 and 375 offshore New Zealand identified hydrate ∼60 m beneath the BSR. This hydrate appears to be dissociating over thousands of years following a gradual temperature increase from sediment burial modulated by changes in bottom‐water temperature and sea‐level fluctuations. Slow hydrate dissociation significantly buffers the release of methane and therefore, carbon through glacial cycles. Dissociating hydrate beneath the BGHS may also increase estimated global budgets of methane stored in hydrate. 

Autonomous vehicles are becoming increasingly popular, but their reliance on computer systems to sense and operate in the physical world introduces new security risks. In this paper, we show that the location privacy of an autonomous vehicle may be compromised by software side-channel attacks if localization software shares a hardware platform with an attack program. In particular, we demonstrate that a cache side-channel attack can be used to infer the route or the location of a vehicle that runs the adaptive Monte-Carlo localization (AMCL) algorithm. The main contributions of the paper are as follows. First, we show that adaptive behaviors of perception and control algorithms may introduce new side-channel vulnerabilities that reveal the physical properties of a vehicle or its environment. Second, we introduce statistical learning models that infer the AMCL algorithm's state from cache access patterns and predict the route or the location of a vehicle from the trace of the AMCL state. Third, we implement and demonstrate the attack on a realistic software stack using real-world sensor data recorded on city roads. Our findings suggest that autonomous driving software needs strong timing-channel protection for location privacy. 

During Expedition 386, two Giant Piston Corer (GPC) system deployments in Basin C/N3 in the boundary area between the central and northern Japan Trench (Figure F1) resulted in the recovery of cores from four holes at Site M0087 (Figure F2). The water depth was between 7518 and 7520 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Site M0087 were cored during Weeks 3 and 6 of the offshore phase. In total, 47.63 m of cores (Table T1) and 69 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired, respectively, in this focus area. In addition, one expendable bathythermograph (XBT) probe was deployed. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, two Giant Piston Corer (GPC) system deployments in the northern study area (Basin S3) of the southern Japan Trench (Figure F1) resulted in the recovery of cores from four holes at Site M0091 (Figure F2). The water depth was between 7802 and 7812 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Site M0091 were cored during Week 6 of the offshore phase. In total, 51.94 m of cores (Table T1) and 53.5 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired, respectively, in the focus area. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material). 

During Expedition 386, one Giant Piston Corer (GPC) system deployment at this central study area of the northern Japan Trench (Basin N1) (Figure F1) resulted in the recovery of cores from two holes at Site M0086 (Figure F2). The water depth was 7502 meters below sea level (mbsl). A breakdown of operational time is reported weekly instead of daily (see OPS in Supplementary material) due to decisions to move between sites based on weather and current conditions. Holes at Sites M0086 were cored during Week 3 of the offshore phase. In total, 19.275 m of cores (Table T1) and 6.65 km of hydroacoustic profiles (see Hydroacoustics) were recovered and acquired in this focus area. An expendable bathythermograph (XBT) probe was deployed at Site M0086 on 2 May 2021 at 0715 h. Further operations details, including winch log and inclinometer information, are found for all sites in Coring methodology in the Expedition 386 methods chapter (Strasser et al., 2023a) and associated files (see PALEOMAG and WINCHLOGS in Supplementary material).